Skip to content

fix(recovery): complete short reads and check offsets in deep walk - #22

Merged
farhan-syah merged 1 commit into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/recovery-diagnostic-integrity
Jul 27, 2026
Merged

fix(recovery): complete short reads and check offsets in deep walk#22
farhan-syah merged 1 commit into
NodeDB-Lab:mainfrom
presempathy-awb:codex/upstream/recovery-diagnostic-integrity

Conversation

@presempathy-awb

@presempathy-awb presempathy-awb commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Deep walk mistook two legal conditions for corruption and computed byte offsets from untrusted counts without checking them. This rebases onto current main and fixes both, with the diagnostics an fsck surface is supposed to produce.

  • A positive short read is not a defect. Every PageDB VFS may satisfy a request across several read_at calls, so deep walk reporting short read: expected 4096 bytes, got 4095 was a false corruption report about a backend that was still making progress. Main-database and segment data-page reads now go through the repository's shared read_exact_at, which retries until the buffer is full and fails only when the backend stops advancing.
  • Offsets derived from disk are now checked. page_id * page_size and page_count * page_size came from catalog records, free-list entries, and header fields, and could wrap in release or panic in debug. Both now go through a checked helper and surface as a structured page or segment issue.

No new dependencies, public APIs, feature flags, durable fields, or format changes.

Not losing the numbers

read_exact_at absorbs legal short reads, which means a genuinely truncated file arrives as a bare UnexpectedEof with no counts attached. On a diagnostic surface those counts are the product, so describe_page_read_failure restores them: a truncated page is reported as truncated: page starts at offset 20480, file is 20480 bytes rather than an opaque read error. Any other error is already self-describing and passes through unchanged.

Shared offset helper

page_offset lives in pager::page_space, the module that already owns main.db's page-id space, and both pager::core::read_page and deep walk now use it instead of open-coding the same checked multiplication. The helper names its caller in the resulting error, because a wrapped offset and a rejected one are indistinguishable by the time a report renders them.

The one-byte probe at a segment's expected end deliberately keeps raw read_at. It asks whether extra data exists past the end; exact-read semantics would change the question.

Scope

Ordinary reads, writes, commits, and segment reads do not touch the changed code. The read change affects the explicit deep-walk/fsck surface only. Healthy behavior is unchanged: against a backend that fills a buffer in one call the page reads issue exactly the same I/O as before, and a backend that returns a legal short read now performs only the extra call needed to finish the transfer.

Tests

  • Two regressions drive a MemVfs wrapper that shortens exactly one read_at by one byte and leaves the missing byte for the next call — for a main-database page and for a segment page. Both fail on main with the false short read issue and pass here. Each asserts the injected short read actually reached deep walk, so neither can pass vacuously.
  • A truncated main database must still be reported, and the assertion pins the offset and file length rather than just the presence of an issue.
  • page_offset is covered directly, including the first page id whose offset is unrepresentable.
  • An impossible catalog page_count must become a structured issue and must not abort the walk. The test also asserts the record is rejected by authenticated-metadata validation before any offset arithmetic runs, so the layering is pinned rather than assumed.
  • sweep_orphans gains its first direct coverage: an expected live segment survives, an unnamed live segment becomes a tombstone rather than a deletion, and an unnamed staging file is removed.
  • The tombstone journal actions gain the cases they were missing — an existing destination must be left byte-for-byte intact, and a live-to-tombstone rename must move the content.

A tombstone's goal state is that the live file is gone, so its absence is itself proof the action completed — unlike a promote, whose goal state is a file that exists. The two are not symmetric, and seg/.tombstone/ is reclaimed wholesale by tombstone GC, which makes "live gone and tombstone gone" the ordinary state of a delete that finished and was then collected. Requiring a surviving tombstone as proof would make replay of an already-completed journal fail permanently and would diverge from the pin-aware Db::tombstone_segment that owns this in production. That reasoning is now recorded next to the test so it is not "fixed" into a fail-closed regression later.

Verification

Rebased onto d8ff86a.

cargo fmt --all --check
cargo clippy --all-targets --all-features -- -D warnings   # warning-free
cargo nextest run --all-features --no-fail-fast            # 713 passed, 10 skipped, 0 failed

Both new deep-walk regressions were re-run against unmodified main production code to confirm they fail for the intended behavioral reason, not on a fixture assertion.

page_id and page_count values reach the pager and deep walk from
on-disk catalog and free-list records, so a corrupted record's count
could overflow the page_id * page_size multiplication and silently
wrap to a bogus offset rather than failing safely. Extract the checked
multiplication pager/core.rs already performed into a shared
page_space::page_offset helper, and route deep_walk's main-page and
segment-page offset computations through it so an unrepresentable
count now surfaces as a structured page or segment issue instead of
wrapping.

deep_walk also moves its page reads onto read_exact_at, which absorbs
legal short reads transparently but collapses a genuine truncation
into a bare UnexpectedEof; add describe_page_read_failure to turn that
back into a diagnostic naming the offset and actual file length.

Expand recovery and fsck test coverage to match: a deep-walk test for
an impossible catalog page_count, a sweep_orphans test covering live,
orphaned-live, and staged-orphan segments in one pass, journal-replay
tests for the tombstone action's idempotent/resume/rename outcomes,
and a ShortReadVfs fixture proving deep walk tolerates a legal short
read on the underlying VFS.
@farhan-syah
farhan-syah force-pushed the codex/upstream/recovery-diagnostic-integrity branch from d759bee to a806aef Compare July 27, 2026 11:37
@farhan-syah farhan-syah changed the title fix(recovery): harden replay and deep-walk reads fix(recovery): complete short reads and check offsets in deep walk Jul 27, 2026
@farhan-syah
farhan-syah merged commit 74b1b37 into NodeDB-Lab:main Jul 27, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants